JFB: Jacobian-Free Backpropagation for Implicit Networks

A promising trend in deep learning replaces traditional feedforward networks with implicit networks. Unlike networks, solve a fixed point equation to compute inferences. Solving for the varies complexity, depending on provided data and an error tolerance. Importantly, may be trained memory costs stark contrast whose requirements scale linearly depth. However, there is no free lunch --- backpropagation through often requires solving costly Jacobian-based arising from function theorem. We propose Jacobian-Free Backpropagation (JFB), fixed-memory approach that circumvents need equations. JFB makes faster train significantly easier implement, without sacrificing test accuracy. Our experiments show are competitive prior given same number of parameters.